To What Extent Is Stress Testing of Android TV Applications Automated in Industrial Environments?

摘要

An Android-based smart Television (TV) must reliably run its applications inan embedded program environment under diverse hardware resource conditions.Owing to the diverse hardware components used to build numerous TV models, TVsimulators are usually not high enough in fidelity to simulate various TVmodels, and thus are only regarded as unreliable alternatives when stresstesting such applications. Therefore, even though stress testing on real TVsets is tedious, it is the de facto approach to ensure the reliability of theseapplications in the industry. In this paper, we study to what extent stresstesting of smart TV applications can be fully automated in the industrialenvironments. To the best of our knowledge, no previous work has addressed thisimportant question. We summarize the find-ings collected from 10 industrialtest engineers to have tested 20 such TV applications in a real productionenvironment. Our study shows that the industry required test automationsupports on high-level GUI object controls and status checking, setup ofresource conditions and the interplay between the two. With such supports, 87%of the industrial test specifications of one TV model can be fully automatedand 71.4% of them were found to be fully reusable to test a subsequent TV modelwith major up-grades of hardware, operating system and application. Itrepre-sents a significant improvement with margins of 28% and 38%,respectively, compared to stress testing without such supports.